DIPLOMARBEIT Support Vector Machines for Regression Estimation and their Application to Chaotic Time Series Prediction

Support vector machines (SVMs) are a quite recent supervised learning approach towards function estimation. They combine several results from statistical learning theory, optimisation theory, and machine learning, and employ kernels as one of their most important ingredients. The present work covers the theory of SVMs with emphasis on SVMs for regression estimation, and the problem of chaotic time series prediction. It is organised in three parts. In the first part, the building blocks that contribute to the theory of SVMs are introduced. The necessary results from statistical learning theory, optimisation theory and kernels are summarised in a modular and self-contained way that makes them accessible as well to readers without background in these topics. The exposition complements already existing material on SVMs in so far as introductory literature that covers all theory employed by SVMs is hard to find. By viewing the function estimation problem as a learning problem, results from statistical learning theory allow the construction of linear learning machines. An application of Lagrangian theory casts these learning machines in forms that constitute the support vector algorithms for classification and regression estimation, which can employ rich classes of nonlinear modelling functions via the use of kernels. We consider in detail the derivation of the standard support vector algorithms for regression estimation and, as an example for a recently reported extension to the standard algorithms, describe the ν-SVM which is capable of tuning one of the parameters involved in support vector training as a part of the training procedure. Then, the connection between maximum likelihood estimation and the choice of the loss function is established, and the introduction of kernels allows us to show how SVMs are related to other function estimation approaches. We finish the first part with a discussion on the important question of how the hyperparameters involved in the support vector algorithms can be assessed. The second part considers the application of SVMs to chaotic time series prediction. The properties of chaos as a feature of nonlinear deterministic dynamical system are reviewed, and it is discussed how observed chaotic data can be analysed via phase space reconstruction and time delay embedding. Practical methods for time delay embedding are reviewed, and we show how SVMs can be applied to the function approximation problem arising from the task of predicting chaotic time series from embedded data. In the third part, the prediction procedure with SVMs is described in detail. The setup of the numerical experiments and an analysis of the time series considered therein are given. We report and illustrate new results for chaotic time series prediction on the Hénon time series, the Mackey-Glass time series, the Lorenz time series and the Santa Fe data set A, obtained with SVMs as global and local models employing different kernel functions. We provide an exhaustive comparison with results reported by other authors. The work finishes with a discussion of the numerical experiments.